Three dimensional tactile feedback system

ABSTRACT

A three dimensional tactile feedback includes a two dimensional array of ultrasonic transducers and a control device configured to control the ultrasonic transducers. The ultrasonic transducers re configured to project discrete points of tactile feedback in three dimensional space. The tactile feedback system is configured to continuously switch the discrete points of tactile feedback at a frequency at which a human is capable of perceiving tactile stimulation in order to produce tactile stimulation of a three dimensional object in free space.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of PCT Application No.PCT/US2015/040045 filed on Jul. 10, 2015, which claims the benefit ofU.S. Provisional Application No. 62/023,538 filed on Jul. 11, 2014, bothof which are hereby incorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates generally to a three dimensional tactilefeedback system. More specifically, the present disclosure relates to asystem designed to provide a three dimensional tactile feedback byemploying acoustic radiation pressure.

BACKGROUND

Natural and three dimensional (3D) interfaces are getting popular thesedays. The popularity of 3D interfaces is mainly due to the spread of 3Dcapture, authoring, and display interfaces (such as 3D contentsgeneration technologies, 3D authoring tools, 3D display interfaces,hologram technology). Adding 3D haptic feedback to these interfaces isthe next natural step to achieve a natural and intuitive human computerinteraction.

A tactile interface displays information using the sense of touch suchas shape, surface texture, roughness, and temperature onto the humanskin. Tactile displays have been proposed in a wide spectrum ofapplications, including virtual reality, Tele-operation, interpersonalcommunication, entertainment and gaming, military, and health care.

Ultrasonic noncontact tactile display is based on the phenomenon ofacoustic radiation pressure. When the focused ultrasound beam isreflected by the surface of an object, the surface is subjected to aconstant force in the direction of the incident beam.

Previous research related to tactile displays can be divided into threecategories: wearable display, touch screen display, and non-touchdisplay. Wearable tactile display embeds vibrotactile actuators inwearable devices to make a direct contact with the human skin. The touchscreen display has vibrotactile actuation technology integrated in thevisual/auditory display interface (such as the touch screen) anddisplays tactile information when the user makes a contact with thetouch screen. Non-touch tactile display uses wireless means to stimulatetactile sensation on the human skin (such as focused ultrasound waves).

Most of the current tactile display devices are of the wearable displayclass. The iFell_IM system is a wearable interface designed to enhanceemotional immersion in a virtual world called Second Life™. Three groupsof haptic gadgets are built. The first group is intended for implicitemotion elicitation (e.g., HaptiHeart, HaptiButterfly, HaptiTemper, andHaptiShiver), the second group functions in a direct way (e.g.,HaptiTickler), and the third group uses a sense of social touch (e.g.,HaptiHug) for influencing the mood and providing some sense of physicalco-presence

A touch panel (screen) interface enables a user to manipulate graphicaluser interfaces through tactile stimulation to strengthen theintuitiveness and directness of interaction. There are commerciallyavailable touch screen devices with tactile interaction capabilities(e.g., “TouchSense”™ from Immersion and “Sensegs Tixel”™ from Senseg),as well as research prototypes. For instance, in one research prototype,a lateral-force-based 2.5-dimensional display was introduced and it wasdetermined that users' impressions of “press”, “guide”, and “sweep” areessential features for touchscreen.

Lately there has been extensive research conducted into using tactiledisplays based on ultrasound waves due to their ability to alter thedefault feeling of touch for the user. One such example is a hybriddisplay using an ultrasonic vibrator and force feedback. It provides arealistic texture sensation, where multiple factors such as roughness,softness and friction are accounted for while taking care of theinterference among the parts. User experience shows high correlationbetween real materials and their hybrid artificial touch counterparts.Another ultrasound tactile display is based purely on amplitudemodulated ultrasonic vibration, where the aim is to replicate thesensation of an ordinary piece of textile using the aforementionedtechnique. The experimental results show that the artificial sensationhighly correlates with its natural version according to test subjects.

A contactless touch screen with tactile feedback that stimulates tactilesensation 1-3 cm in front of the screen surface has been proposed. Asingle point of tactile stimulation is generated using an airborneultrasound phased array.

Air jets are also utilized to create non-contact force feedback wherefans or air cannons are used in theme parks to amaze visitors. Althoughthey are effective in simulating a rough “force” feedback, theirspatial/temporal properties are limited and they cannot provide finetactile feedback.

A more convenient approach in designing non-touch displays is usingairborne ultrasound. A tactile display has been presented that utilizedairborne ultrasound to produce tactile sensations with 16 mN at avirtual point in space. The demonstrated prototype has a spatialresolution of 20 mm and produces vibrations up to 1 kHz. Experimentsshowed that users could identify the tactile stimulus generated by thedevice and were able to discriminate its moving direction. A similarwork that utilized a two dimensional array of ultrasonic transducers togenerate concentrated pressure at an arbitrary point in a largerworkspace has also been developed. This multi-point haptic feedbacksystem (e.g., UltraHaptics) is used over an interactive surface that iscapable of producing independent feedback points in relation toon-screen elements. However, the UltraHaptics system is limited to twodimensional surface display.

Yoshino discloses a contactless touch interface is demonstrated that isdeveloped with blind people as its target audience. See Yoshino, K. etal., “Contactless touch interface supporting blind touch interaction byaerial tactile stimulation,” 2014 IEEE Haptics Symposium (HAPTICS), pp.347-350, 2014, the entire contents of which is incorporated by referencein its entirety. The system consists of two separate sensible layers forguidance and operation, respectively. Tactile feedback is also part ofthe system. A user study is conducted with high success rate and anaverage time of 6.5 seconds for finding the next button.

The exact ultrasonic pressure field created by ultrasonic actuators isalso a relevant field of research. Hoshi presents an approach formeasuring the strength of the ultrasonic pressure field. See Hoshi, T.et al., “Non-contact tactile sensation synthesized by ultrasoundtransducers,” Third Joint Symposium on Haptic Interfaces for VirtualEnvironment and Teleoperator Systems, pp. 256,260, 2009, the entirecontents of which is incorporated by reference in its entirety. Theapproach achieved a spatial resolution of less than 0.1 mm with aminimal detectable pressure of 50 kPa (compared to atmospheric pressure)on a 2-D cross-sectional area of 15 mm by 5 mm. Furthermore, Svein-ErikMasoy explores the overall radiation pressure field of a systemcomprising of two ultrasonic transducers with different radiationfrequencies. See Svein-Erik Masoy, Thor Andreas Tangen et al.,“Nonlinear propagation acoustics of dual-frequency wide-band excitationpulses in a focused ultrasound system”, Journal of the AcousticalSociety of America, vol. 128, no. 5, 2010, the entire contents of whichis incorporated by reference in its entirety. Simulation results closelyagree with experimental measurements and the investigators conclude witha number of considerations for designing dual-frequency ultrasoundsystems.

Two dimensional tactile feedback systems, which construct twodimensional objects, have been described in the prior art. For example,Hoshi discloses a compact that enables users not only to operatecomputers by moving their hands in mid-air, but also to feel hapticfeedback on their hands. See Hoshi, Takakyuki, “Compact Device forMarkerless Hand Tracking and Noncontact Tactile Feedback,” SICE AnnualConference 2013, Nagoya University, Nagoya, Japan (Sep. 14-17, 2013),the entire contents of which is incorporated by reference in itsentirety. The device consists of a depth camera for hand tracking and anoncontact tactile display for haptic feedback. See id. The tactiledisplay utilizes focused ultrasound to produce tactile stimulation froma distance. See id. Further background information on tactile displaysfor transmission of sensory data to a human by an acoustic method basedon the effect of radiation pressure can be found in Gavrilov, L. R.,“The Possibility of Generating Focal Regions of Complex Configurationsin Application to the Problems of Stimulation of Human ReceptorStructures by Focused Ultrasound,” Acoustical Physics, Vol. 54, No. 2,pages 269-278 (2008), the entire contents of which is incorporated byreference in its entirety.

The devices mentioned do not produce tactile sensations of threedimensional objects by way of airborne ultrasound. A need exists forimproved technology that generates acoustic radiation forces at pointsin 3D space and constructs a 3D object.

SUMMARY

Various embodiments relate to a three dimensional tactile feedbackdevice. The tactile feedback device comprises a two-dimensional array ofultrasonic transducers and a control device configured to control theultrasonic transducers. The ultrasonic transducers are configured toproject discrete points of tactile feedback in three dimensional space.The tactile feedback device is configured to continuously switch thediscrete points of tactile feedback at a frequency at which a human iscapable of perceiving tactile stimulation (such as 1 KHz) in order toproduce tactile stimulation of a three dimensional object in free space.

Other embodiments relate to a method of producing a tactile stimulationof a three dimensional object. The method comprises providing a threedimensional tactile feedback device comprising a two dimensional arrayof ultrasonic transducers and a control device configured to control theultrasonic transducers, projecting discrete points of tactile feedbackin three dimensional space with the ultrasonic transducers, andcontinuously switching the discrete points of tactile feedback at afrequency at which a human is capable of perceiving tactile stimulation(such as 1 KHz) in order to produce tactile stimulation of a threedimensional object in free space.

Further embodiments relate to a computer-implemented machine for threedimensional tactile feedback comprising a processor and a threedimensional tactile feedback device. The three dimensional tactilefeedback device is comprised of a two-dimensional array of ultrasonictransducers, a control device configured to control the ultrasonictransducers, and a tangible computer-readable medium operativelyconnected to the processor. The tangible computer-readable mediumincludes computer code configured to project discrete points of tactilefeedback in three dimensional space with the ultrasonic transducers, andcontinuously switch the discrete points of tactile feedback at afrequency at which a human is capable of perceiving tactile stimulationin order to produce tactile stimulation of a three dimensional object infree space.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features and aspects of thesubject matter will become apparent from the description, the drawings,and the claims presented herein.

FIG. 1 is a schematic illustration of a three dimensional tactilefeedback device (i.e., a Haptogram system).

FIG. 2 is an in-depth schematic illustration of the Haptogram system ofFIG. 1.

FIG. 3 is a Graphical User Interface of the Haptogram system of FIG. 1,showing a point cloud for various 3D tactile objects.

FIG. 4 is a field-programmable gate array (FPGA) block diagram design ofthe Haptogram system of FIG. 1.

FIG. 5 is an ultrasound transducer amplification circuit of theHaptogram system of FIG. 1.

FIG. 6A is a top view of a schematic illustration of a single tile,two-dimensional 10 by 10 array of ultrasound transducers of theHaptogram system of FIG. 1.

FIG. 6B is an isometric view of the tiled two-dimensional 10 by 10 arrayof ultrasound transducers of FIG. 6A

FIG. 7 is a tiled two-dimensional array design using the tiles of FIG.6A.

FIG. 8 is a flowchart for calculating a focal point using the Haptogramsystem of FIG. 1.

FIG. 9 is an experimental setup used to measure the spatial distributionof the acoustic radiation pressure of the Haptogram system of FIG. 1.

FIG. 10 illustrates the theoretical location of the single focal pointstimulated at 12 cm elevation (scale in m).

FIG. 11 illustrates the spatial distribution of the acoustic radiationforce around a focal point that is generated at 13 cm elevation from theultrasound array surface.

FIG. 12 is a graph illustrating the force versus distance for focalpoint stimulation for a series of focal points starting from anelevation of 8 cm up to 17 cm.

FIG. 13 illustrates a point cloud distribution of focal points for astraight line stimulated at 12 cm elevation (scale in m).

FIG. 14 illustrates the spatial distribution of the acoustic radiationforce for the straight line of FIG. 13 that is generated at 12 cmelevation from the ultrasound array surface.

FIG. 15 illustrates a point cloud distribution of focal points for acircle stimulated at 12 cm elevation (scale in m).

FIG. 16 illustrates the spatial distribution of the acoustic radiationforce for the circle of FIG. 15 that is generated at 12 cm elevationfrom the ultrasound array surface.

FIG. 17 illustrates a point cloud for a hemisphere object centered at 14cm elevation (scale in m).

FIG. 18 illustrates the spatial distribution of the acoustic radiationforce for the hemisphere object of FIG. 17 that is centered at 14 cmelevation from the ultrasound array surface.

FIG. 19 is a simulation rendering of a 3D tactile object stimulation.

FIG. 20A is a simulation rendering of a 10 mm resolution tactile 3Dobject.

FIG. 20B is a simulation rendering of a 5 mm resolution tactile 3Dobject.

FIG. 20C is a simulation rendering of a 2 mm resolution tactile 3Dobject.

FIG. 21 is a simulation rendering of desired tactile feeling points.

FIG. 22 is a schematic illustration of how a Haptogram system will beused in immersive multimodal interactive systems.

FIG. 23 illustrates a coordinate system with a 2D ultrasound array ofM×N transducers and an arbitrary focal point F_(P).

FIG. 24 illustrates an example of a pulse generator output voltagewaveform.

FIG. 25 illustrates a focal point formation delay in a voltage waveformmeasured by an ultrasound receiver probing for acoustic pressure at 15cm elevation over time.

FIG. 26 illustrates four 2D shapes (circle, triangle, line, and plussign) utilized in a usability study to investigate how well usersperceive animated 2D shapes displayed by the Haptogram system.

FIG. 27 is a graph showing an average recognition rate, along with astandard deviation, for each shape illustrated in FIG. 26.

FIG. 28 is a graph showing an average recognition time, along with astandard deviation, for each shape illustrated in FIG. 26.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Before turning to the figures, which illustrate the exemplaryembodiments in detail, it should be understood that the presentapplication is not limited to the details or methodology set forth inthe description or illustrated in the figures. It should also beunderstood that the terminology is for the purpose of description onlyand should not be regarded as limiting.

Ultrasonic tactile stimulation involves focusing multiple ultrasoundbeams in one or more focal points to produce tangible mid-air acousticpressure effect. When a transducer array is driven such that the phasesof the ultrasonic waves coincide at a point (i.e., a focal point), theradiation pressure will be strong enough to be perceived by human skin.Embodiments of a Haptogram system that provides 3D tactile feedback viafocused ultrasound, and requires no physical contact with the human skinare described below. The Haptogram system described herein renderspoint-cloud 3D tactile objects and produces 3D tactile sensation byanimating focal points at high speed (switching speed) so that humansfeel the distributed points simultaneously. Visual rendering generatesgraphics to be displayed by a hologram display where haptic renderingcalculates tactile forces to be displayed by the Haptogram system. Thetactile sensations are displayed by generating acoustic radiation forcesat focal points where a phased array of ultrasonic transducers is usedto exert forces on a target point in 3D space to construct a 3D object.The system continuously switches tactile display points at a very highspeed (1 kHz update rate) to display tactile sensation of the 3D objectin free space. The successive focal points altogether make the 3Dtactile object. Moving the point of tactile stimulation at very highspeed along a 3D model creates a 3D tactile experience.

The tactile feedback device is configured to continuously switch thediscrete points of tactile feedback at a frequency at which a human iscapable of perceiving tactile stimulation (such as 1 KHz) in order toproduce tactile stimulation of a three dimensional object in free space.Humans are capable of perceiving vibrations in a frequency range of 40Hz to 800 Hz. In one embodiment, the tactile feedback device isconfigured to continuously switch the discrete points of tactilefeedback at a frequency in the range of 10 Hz to 5 KHz. In an additionalembodiment, the tactile feedback device is configured to continuouslyswitch the discrete points of tactile feedback at a frequency in therange of 40 Hz to 5 KHz. In another embodiment, the tactile feedbackdevice is configured to continuously switch the discrete points oftactile feedback at a frequency in the range of 40 Hz to 800 Hz. In yetanother embodiment, the tactile feedback device is configured tocontinuously switch the discrete points of tactile feedback at afrequency in the range of 125 Hz to 300 Hz.

The resolution of the tactile 3D object may be varied by producing ahigher or lower number of successive focal points. Producing higherresolution may take more time to display each of the points and thuswould require faster hardware to implement. A higher resolution wouldprovide a higher quality of user perception of the 3D tactile object.

The tactile stimulation produced by the Haptogram system is based on thePhased Array Focusing technique to produce the radiation pressureperceivable by the human skin. The focal point of ultrasound—a set ofthese points make up a 3D shape—is generated by controlling the phasedelays of multiple transducers at a high update rate. When N transducersare driven so that the phases of the ultrasounds coincide at a point,the radiation pressure that is generated by the array will be largeenough to be perceivable by human skin.

The theoretical derivation of the resulting sound pressure field on the3D surface tactile points is now described. First, the specifications ofthe transducer array are listed. The diameters of the transducer housingand the diaphragm are d=10 mm and 8 mm, respectively. The resonantfrequency is 40 KHz whereas the directivity is 80 deg.

Secondly, the sound pressure field is formulated. Let r[m] be the vectorfor a general transducer on the array (x_(m), y_(m), 0) and a generalfocal point with coordinates (x₀, y₀, z₀) on the Haptogram. The RMSsound pressure p₀ from that transducer onto the focal point is inverselyproportional to the square of (r). Therefore, the resulting soundpressure field P(x₀, y₀, z₀) is written as shown in equation (1).Equations (3) to (6) derive an expression for the acoustic radiationpressure (P) as function of the elevation (z). The final expression ofthe pressure field becomes as shown in equation (7).

$\begin{matrix}{{P\left( {x_{0},y_{0},z_{0}} \right)} = {\sum\limits_{m = 0}^{M - 1}{\sum\limits_{n = 0}^{N - 1}{\sqrt{2}{P\left( z_{0} \right)}^{j{({{kr} - {wt}})}}}}}} & (1) \\{r = \sqrt{\left( {x_{m} - x_{0}} \right)^{2} + \left( {y_{m} - y_{0}} \right)^{2} + \left( z_{0} \right)^{2}}} & (2) \\{{P\left( z_{0} \right)} = {P_{0}^{{- 2}{\beta z}_{0}}}} & (3) \\{E = {\frac{1}{C} = \frac{P^{2}}{\rho \; C^{2}}}} & (4) \\{{E\left( z_{0} \right)} = {E_{0}^{{- 2}{\beta z}_{0}}}} & (5) \\{{P(z)} = {\sqrt{E\; \rho \; C^{2}} = {{\sqrt{E_{0}\rho}C\; ^{{- \beta}\; z}} = {P_{0}^{{- \beta}\; z}}}}} & (6) \\{{P\left( {x_{0},y_{0},z_{0},t} \right)} = {\sum\limits_{m = 0}^{M - 1}{\sum\limits_{n = 0}^{N - 1}{\sqrt{2}P_{0}^{{- \beta}\; z_{0}}^{j{({{kr} - {wt}})}}}}}} & (7)\end{matrix}$

In the equations above, M and N are the dimensions of the ultrasoundarray, β is the attenuation coefficient of the air interface, k is thewavenumber of the ultrasound signal, w is the angular frequency (2πf), fis the resonance frequency for the ultrasound transducers, and r is thefocal length.

Equation (8) determines the size of the focal point and the total forceF [N] contained within a cubic unit of side πf (geometrically derived tobe as expressed in equation (9).

$\begin{matrix}{F = {\frac{\alpha}{\rho \; c^{2}}{\int_{{- \omega_{f}}/2}^{{+ \omega_{f}}/2}{\int_{{- \omega_{f}}/2}^{{+ \omega_{f}}/2}{\int_{{- \omega_{f}}/2}^{{+ \omega_{f}}/2}{\frac{{{p\left( {x_{0},y_{0},z_{0}} \right)}}^{3}}{3}{x_{0}}{y_{0}}{z_{0}}}}}}}} & (8) \\{Where} & \; \\{\omega_{f} = \frac{4\pi \; r}{k\sqrt{MNd}}} & (9)\end{matrix}$

Acoustic Principles and Modeling

Consider the coordinate system shown in FIG. 23, with a 2D ultrasoundarray of M×N transducers and an arbitrary focal point F_(P) shown inequation (10). The distance between the focal point and each transduceris divided by the speed of sound to find the ultrasound wave travel timet_(mn) for each transducer as shown in equation (11), so that theultrasound waves come in phase at the focal point to generate acousticpressure. In equation (10), R is the distance from the origin to thefocal point, θ is the incident angle, and φ is the rotational angle. Thefocal law can be obtained from equation (12), and thus the transducersdelays are calculated via equation (14).

F _(P)=(R sin θ cos φ, R sin θ sin φ, R cos θ)   (10)

Then for any transducer E_(mn) at (x_(m), y_(m), 0):

$\begin{matrix}{t_{mn} = \frac{{F_{p} - E_{mn}}}{c}} & (11) \\{T_{mn} = {{\max \left( t_{mn} \right)} - t_{mn}}} & (12) \\{{\Delta \; T} = {{\max \left( t_{mn} \right)} - {\min \left( t_{mn} \right)}}} & (13) \\{{\Delta \; T_{mn}} = {\left\lbrack T_{mn} \right\rbrack {{mod}\left\lbrack \frac{\lambda}{c} \right\rbrack}}} & (14)\end{matrix}$

Where [ ] denote the integer part in equation (14), λ the wavelength, cthe sound speed, and ΔT_(mn) is the time shift per transducer in orderfor all of them be in phase at the focal point. The pressure from atransducer E_(mn), assuming a point source modeling, can be described asan outgoing spherical wave using equation (15), where A is the maximumpressure, R is the distance from the transducer to the focal point, ωthe angular frequency, t is the time, and k is the wave number.

$\begin{matrix}{{p\left( {R,t} \right)} = {\frac{A}{R}^{j{({{\omega \; t} - {kR}})}}}} & (15)\end{matrix}$

For an array of transducers M*N, the pressure contributed by eachtransducer p_(mn) (R_(mn), t_(mn)) is calculated as shown in equation(16), where m, n=1 . . . M, N. R_(mn) is the distance between transducerE_(mn) and the focal point, whereas t_(mn) is the time it takes theultrasound wave to travel from transducer E_(mn) to the focal point.

$\begin{matrix}{{p_{mn}\left( {R_{mn},t_{mn}} \right)} = {\frac{A}{R_{mn}}^{j{({{\omega \; t_{mn}} - {kR}_{mn}})}}}} & (16)\end{matrix}$

where R_(mn)=√{square root over ((x₀−x_(m))²+(y₀−y_(m))²+z₀ ²)}. Thetotal pressure can be derived as the superposition of the pressuregenerated by each transducer at the focal point F_(P)=(x₀, y₀, z₀) byusing the focal law. The resulting formula for the pressure field isgiven by equation (17).

$\begin{matrix}{{p\left( {R,\theta,t} \right)} = {\sum\limits_{mn}^{MN}{\frac{A}{R_{mn}}^{j{({{\omega {({t + {\Delta \; T} + {\Delta \; t_{mn}}})}} - {kR}_{mn}})}}}}} & (17)\end{matrix}$

A more precise modeling is to represent the transducer by a circularsurface rather than a point source. Therefore, the transducer E_(mn) canbe replaced by the far field approximation of the Rayleigh-Sommerfeldintegral model with a circular surface (piston) of radius α, as shown inequation (18).

$\begin{matrix}{{p_{0}\left( {R,\theta,t} \right)} = {{j\omega\rho}\; U_{0}{\alpha^{2} \cdot {\sum\limits_{mn}^{MN}{\frac{A}{R_{mn}}{^{j{({{\omega {({t + {\Delta \; T} + {\Delta \; t_{mn}}})}} - {kR}_{mn}})}}\left\lbrack \frac{J_{1}\left( {k\; \alpha \; \sin \; \theta_{mn}} \right)}{k\; \alpha \; \sin \; \theta_{mn}} \right\rbrack}}}}}} & (18)\end{matrix}$

where

$\theta_{mn} = {\tan^{- 1}\left( \frac{\sqrt{\left( {x - x_{m}} \right)^{2} - \left( {y - y_{m}} \right)^{2}}}{z} \right)}$

is the angle between R_(mn) and the axis perpendicular to transducersplane, whereas J₁ is the first order Bessel function, U₀ is the uniformparticle velocity generated by a circular surface of radius α, and ρ isthe medium density. The far field approximation is accurate fordistances that satisfy

${R\operatorname{>>}\frac{d^{2}}{4\lambda}},$

where d is 2α.

Haptogram System Overview

In general, a three dimensional tactile feedback device (i.e., aHaptogram system 1000) is comprised of a software subsystem 100 and ahardware subsystem 200, as illustrated in FIGS. 1 and 2. As seen in FIG.1, an Input Devices component acquires position and intensity knowledgeabout a three dimensional model through the Position Input and IntensityInput units, respectively. The software subsystem 100 is programmed toperform at least the following functions: 1) distance calculation tocompute distance between the three dimensional model and a twodimensional ultrasound array, 2) timing calculation to determine phaseshifts for the ultrasonic transducers to produce a three dimensionalshape, and 3) signal generation to compose electric signals to feed theultrasonic transducers. The hardware subsystem 200 has a signalamplifier to produce the intended intensity for the tactile stimulationand a driver circuit that feeds the synchronized control signals to theultrasonic transducer array. The ultrasonic transducer array can beconfigured by physical dimension (e.g., a single dimensional array or atwo dimensional array) and layout (such as linear, circular, randomlayout, etc.).

As seen in FIG. 2, the software subsystem 100 has three components: aGraphical User Interface (GUI) 110, a 3D Point Cloud Representation(PCR) component 120 and a Focal Point Calculation (FPC) component 130.The GUI 110 is similar to a graphical authoring tool that provides theend user with a standard interface to create tactile objects that can bedisplayed using the Haptogram system (single point, 2D or 3D objects).The user would author/select from a list of existing tactile objects andload the corresponding PCR 120 representing the selected object into afield-programmable gate array (FPGA) circuit for execution. A snapshotof the GUI is shown in FIG. 3.

The PCR component 120 calculates a finite set of points or 3D pointcloud representations that approximate the authored shape. Each 3D pointcloud representation includes a plurality of discrete points thatcorrespond to a single point, a two dimensional object or a threedimensional object. The PCR component 120 calculates two hundreddiscrete points that approximate the authored shape, along with theorder at which these discrete points must be displayed. Each discretepoint is represented by 3D coordinate (x,y,z) and is sent to the FPCcomponent 130. The user can select a desired single point, twodimensional object or three dimensional object, and the discrete pointsof the 3D point cloud representation corresponding to the selectedsingle point, two dimensional object or three dimensional object areprojected into three dimensional space to produce tactile stimulation ofthe selected object.

The FPC component 130 calculates the distances between each focal pointand the transducers, and the stimulation timings/intervals of eachtransducer to produce the focal point at the desired (x,y,z) location.Each transducer is controlled by a separate control signal. The FPCcomponent 130 stores the results in a hexadecimal file that is loadedinto the primary memory of the FPGA controller.

The hardware subsystem is made up of three components: thefield-programmable gate array (FPGA) Controller 210, the Driver CircuitAmplification (DCA) component 220 and the Ultrasound Array (UA) 230. TheFPGA Controller 210 produces synchronized pulse signals that feed intothe DCA 220. The DCA component 220 is basically an amplifier circuitthat produces enough power to stimulate the ultrasound transducers.Finally, the UA 230 is a composition of tile units comprised oftwo-dimensional arrays of ultrasound transducers that are expandable toincrease a device workspace and an intensity of tactile stimulation.Each tile unit generates the focal point of compressed pressure at thedesired 3D location in a workspace on top of the UA 230. The twodimensional array of ultrasonic transducers may be stationary andlocated on a table (for example). The ultrasonic transducers areactuated to produce a focal point of concentrated pressure at a specificpoint in 3D space. The specific point in 3D space may be located about15 to 25 cm above the array, but may also be located at a point furtheraway or closer depending on the configuration of the ultrasonictransducers.

FPGA Design

The FPGA 210 design, shown in FIG. 4, includes six basic sub-designs:The primary memory 211, the secondary memory 212 (also referred to as asequence memory), the serial interface 213 (also referred to as a UARTport), the memory controller 214, the cycle controller 215 and the pulsegenerator 216.

All the information needed to form a plurality of focal points is storedto the primary memory 211. For example, the primary memory 211 may storeinformation needed to form up to two hundred focal points. However, theprimary memory 211 may store information needed to form any number offocal points, depending on the available memory of the FPGA board, theprocessing power, etc. The details of each focal point includes: thetiming queues for each transducer (e.g., 100 16-bit numbers in total),the cycle length (e.g., period of the pulse, 24-bit), the pulse length(e.g., 8-bit) and a pre-scalar clock divider (e.g., 24-bit) value. Thetiming queue defines when to switch the transducer on/off in referenceto the clock. In the example including 200 focal points, the cyclelength defines the duration for which the entire 200 focal points mustbe displayed (one cycle is a display for the 200 focal points). Thepulse length is the duration of the pulse that contributes to thestimulation of one focal point. The clock divider scales the timings toprovide a wide range of tactile stimulation frequencies (e.g., rangingfrom 10 Hz to 5 KHz, 40 Hz to 5 KHz, 40 Hz to 800 Hz, or 125 Hz to 300Hz). The total sum of 16-bit words is 106 (there are 40 dummy bits whichare discarded). The memory data is accessed via a script that isdeveloped for this purpose (called Tool Command Language or TCL script),through the JTAG interface.

In the secondary memory 212 a sequence of index numbers between 1 and200 is stored. The secondary memory 212 drives the primary memory 211 byusing these index numbers to select the appropriate focal point thatneeds to be formed at a specific time.

The serial interface 213 is configured to drive the primary memory 211and enable or disable the secondary memory 212. Each time the user wantsto change a focal point, the user has to send, through the serialinterface 213, an index number. For example, in the embodiment includingtwo hundred focal points, each focal point is assigned an index numberbetween 1 and 200. Other index numbers (e.g., 201 and above) may be usedfor control. For example, index number 224 (binary 11100000) may be usedas a special code to activate or deactivate the secondary memory 212(which implies that focal point data would be received through theserial port or retrieved from the secondary memory 212).

The memory controller 214 controls the primary memory 211. The memorycontroller 214 extracts all the needed information out of the primarymemory 211 in order to form a single focal point. All the extracted dataare loaded to the cycle controller 215 and the pulse generator 216.

The cycle controller 215 determines the width of the period of the pulseand drives a 24-bit counter with it. The counter output is then fed tothe pulse generator 216 and used to produce the appropriate pulses. Thecycle controller 215 is also driven by the repeat input, which dictatesthe system to perform just once or continuously.

The pulse generator 216 is the most important sub-design of the FPGA210. In this design, all the appropriate pulses are produced to createthe desired focal point. The data from the primary memory 211 and fromthe cycle controller 215 driven counter are fed to the pulse generator216. The timing queue data provided by the primary memory 211 iscompared in each time step with the cycle controller 215 driven countervalue in such a way that the appropriate, separate, pulse for eachtransducer is generated at the appropriate time.

Those pulses are finally fed to the transducer amplification circuit240. An example of the voltage waveform for sample transducers is shownin FIG. 24. As illustrated in FIG. 24, there is a phase shift betweensuccessive transducers to achieve acoustic pressure at the focal point.

Amplification Circuit

The FPGA controller 210 produces the pulses needed for the transducerarray to form the desired focal points. However, before those pulses arefed to the array, they have to be driven through an amplificationcircuit 240 that guarantees maximum power of the produced ultrasonicwaves to produce tangible tactile sensation at the desired focal point.Each pulse (e.g., out of the 100) is driven to the non-inverting inputof an Op-Amp, which is configured as a voltage comparator 241, as shownin the circuit schematics of FIG. 5. A reference voltage is fed to theinverting input of the Op-Amp 241. The maximum voltage supply of theOp-Amp 241 is ±20 Volts. When the signal in the non-inverting input isless than the reference voltage, the output of the Op-Amp 241 drops tonegative power supply (e.g., −20V). When the signal is for, example, 3.3Volts in the non-inverting input, the output of the Op-Amp 241 rises tothe positive power supply (+20V). The output from each Op-Amp 241 isthen driven to the transducer array 230.

Ultrasonic Array

The amplified signals are fed to a tiled two-dimensional array ofultrasound transducers 230. Each display tile, as shown in FIG. 6, is a10 by 10 array of ultrasound transducers that connects to the centralFPGA 210. In one embodiment, the MA40S4S 44 kHz Murata ultrasoundtransducer may be used in the Haptogram system. The software is designedto accommodate N number of tiles, as appropriately defined by theapplication. The more tiles the Haptogram system 1000 has, the largerthe tactile display workspace. Furthermore, the intensity of the tactilestimulation increases as the number of display tiles increases.

The tiled two-dimensional array design, shown in FIG. 7, has severaladvantages. First of all, a tiled display can be arbitrarily scaled toextend the 3D tactile stimulation workspace. Secondly, a tiled displayprovides a flexible display apparatus, increases tactile stimulationintensity, reduces manufacturing difficulties and provides substratespace for wiring and control circuits on the display tile substrate(thereby improving system integration).

Focal-Point Calculation

The software design consists of a basic function that calculates thetimings that are necessary in order to form, at any given point, a focalpoint by the transducers array 230. This function accepts thecoordinates of the desired focal point (x,y,z) and returns a list ofdistances between the given focal point and each transducer of the array230. A flowchart that explains the focal point calculation is shown inFIG. 8.

Once the function is evaluated another function accepts this list andcalculates all the timings needed by each transducer in order to produceultrasound waves, which are in phase with all the ultrasound waves thatare produced from the rest of the transducers for the given focal point.

These two functions can be called at most 200 times, which is themaximum data set that can be uploaded to the FPGA memory.

The outcome is a list of timing data for (at most) 200 focal points.This final list is fed to a function that converts the list to anappropriate hex file using a modified version (for 16 & 32 bit words) ofa library called intelhex.

The last function of the basic script loads the hex file into theprimary memory of the FPGA by calling the TLC script via the quartus_stpexecutable.

The basic script is the core of the application developed to control theultrasound array. The TLC script is the interface script between thesoftware and the FPGA design. The TLC script drives the quartus_stp.exethat can edit the memory contents of the Altera FPGA via the JTAGprogrammer. The TLC script can also edit constants that are hardcoded inthe FPGA fabric. This TLC script uploads the hex files to the FPGAmemory and it is also used for debugging and controlling the FPGA designby editing the constants. On the other hand, the secondary memorycontains a series of numbers (0-200) and it is responsible of drivingthe primary memory. For example if a user loads the sequence 1-2-3-4fifty times (1-2-3-4-1-2-3-4- . . . 1-2-3-4) then the primary memorywill form only the first four focal points for fifty times according tothe sequence.

Tactile Rendering

Given the FPGA controller runs at a frequency f_(P), the number of clockcycles needed to prepare the actuation signal for switching focal pointsis C and the frequency of ultrasound waves is f_(U), the time requiredto produce a focal point is

$\frac{1}{f_{U}} + {\frac{c}{f_{P}}.}$

However, additional time is needed for all transducers to contribute tothe formation of the focal point to generate maximized pressure. Theminimum number of pulses needed for all transducers to contribute to theformation of the focal point is denoted as β and is defined by equation(19), based on equation (13). β is an integer number. If the renderingfrequency for the focal points is f_(R), then the maximum number offocal points that can be generated to create a 2D/3D tactile object (V)is expressed in equation (21).

β=[f _(U)(maxΔt _(ij)−minΔt _(ij))]  (19)

The total time needed to form maximized pressure at the focal point isgiven by equation (20).

$\begin{matrix}{T_{f} = {\frac{\beta}{f_{U}} + \frac{c}{f_{P}}}} & (20)\end{matrix}$

Where T_(f) is a critical for fast switching of focal points and isreferred to as the Focal Point Formation (FPF) delay. This delayincludes the time for the transducer to settle vibration amplitude sincestart of driving, dead time due to time of flight of the ultrasoundwave, and propagation time. Given V as the number of focal points, therendering frequency f_(R) is defined as shown in equation (21).

$\begin{matrix}{V = \frac{\frac{1}{f_{R}}}{\frac{\beta}{f_{U}} + \frac{c}{f_{P}}}} & (21)\end{matrix}$

Equation (21) demonstrates an inversely proportional relationshipbetween the rendering frequency and the number of focal points. Thisimplies that generating higher resolution tactile display requires anincrease in the number of focal points to form the tactile object. Thisin turn results in lower rendering frequency (if maximum acousticpressure per focal point is to be maintained where all transducerscontribute to the focal point formation). On the other hand, in order toincrease the rendering frequency of tactile objects, a decrease in thenumber of focal points (per object) is to be expected. It then becomes auser choice to pick the proper configurations of the Haptogram system tomake specifications for a particular application.

In order to describe this relationship analytically, one implementationof the Haptogram system utilizes a 10×10 array of ultrasonic transducerswith a resonance frequency f_(u)=40 kHz, C=108, and a processor clockf_(P)=50 MHz. Assuming that the center of the display workspace is at anelevation of 10 cm, equation (13) provides a calculation for ΔT=643 μs.The minimum number of pulses for all transducers to contribute to theformation of a focal point is β=26, based on equation (19). Thereforethe relationship between the rendering frequency

$V = {\frac{10^{3}}{0.643\; f_{R}}.}$

If a rendering frequency of 30 Hz is sufficient for a particular tactiledisplay, then a total number of focal points of V=51 can be achievedwhile maintaining maximized acoustic pressure. A larger renderingfrequency can still be achieved with the same number of focal points butwith lower acoustic pressure. As long as humans perceive the generatedacoustic pressure, such alterations remain valid system configurations.

An experiment is conducted to measure the focal point formation delayusing one implementation of the Haptogram system. An ultrasound receiverin used to measure the generated acoustic pressure at 15 cm elevation.FIG. 25 shows the voltage waveform measured by the ultrasound receiverprobing for the acoustic pressure at 15 cm elevation over time.Considering the 90% rise time, the FPF delay is measured to be around640 μs, which is consistent with the analytical result of 643 μs.

Performance Evaluation

The performance of the Haptogram system was evaluated to assess theeffectiveness of tactile simulation using the Haptogram system. Theanalysis focused on the use of a single tile in the Haptogram display.

In order to measure the spatial distribution of the acoustic radiationpressure, the experimental setup shown in FIG. 9 was used. An ultrasonicsensor probe was attached to the end effector of the robotic arm (STRobotics R17) whose resolution for movement was 0.1 mm.

The sound probe was a MA40S4S 44 kHz Murata ultrasound transducer. Itsoutput was fed to AC-to-DC conversion circuit (to a rectifying bridgeand then to an array of decoupling capacitors). Finally, the resultingDC signal (which represents the sound intensity) is fed to a 10-bitAnalog to Digital Converter (ADC). The following configurations werealso adopted in this experiment: a focal point diameter of 10 mm, thecurrent was limited to 900 mA, the voltage is limited to ±12 V, and themodulation frequency is set to 500 Hz.

A simple script was developed to control the robotic arm to scan anadjustable 3D volume on top of the ultrasound array to measure thedistribution of the acoustic radiation forces. Data were acquired atevery 1 mm around the focal point in the case of single pointstimulation and 2 mm for 2D and 3D shapes. At a general position, 20measurements were taken at 1 ms intervals and the average is calculatedand recorded as the force actuation at that position.

Single Point Tactile Stimulation

For the single point tactile stimulation scenario, the scan surface wasadjusted with the following configuration: 50mm by 50 mm, step size of 1mm (giving a total of 2500 measurements for a single slice along thevertical axis). The robotic arm performed 10 slices of measurementsalong the vertical axis that are centered at the tactile stimulationpoint.

Since the force amplitude turned out to be time varying, the averagevalues were recorded. FIG. 10 shows the theoretical location of thefocal point. FIG. 11 shows the spatial distribution of the acousticradiation force around a focal point that is generated at 13 cmelevation from the ultrasound array surface. The amplitude is normalizedwith a maximum force of 2.9 mN. As shown in FIG. 11, one focal point issurely generated and its diameter is about 10 mm. Note that side lobesaround the main lobe are generated that are nearly symmetrical.

Another study was conducted to estimate the optimal elevation of theworkspace for one tile of the Haptogram system. A series of focal pointsstarting from an elevation of 8 cm up to 17 cm were projected and thecorresponding force presented at each elevation was measured, with astep size of 1 cm. The results, as shown in FIG. 12, demonstrate thatmaximum forces (2.9 mN) are generated at an elevation close to 14 cm,which decreases as the elevation increases or decreases. This isjustified by the fact that as the elevation increases (beyond 14 cm),the ultrasound signals experience higher attenuation, which results inlower perceived forces. On the other hand, as the elevation decreasesbelow the 14 cm height, some transducers would have less contributiontowards the formation of the focal point (due to limited directivity ofthe transducers). Therefore, it would be best to have the center of thedevice workspace for one tile at the 14 cm elevation since this willproduce maximum tactile stimulation, and consequently result in betterperception of the displayed tactile stimulus and higher quality of userexperience.

2D Tactile Stimulation

For the 2D tactile stimulation, two 2D shapes, a straight line and acircle, were considered. The scan volume was adjusted with the followingconfiguration: 100 mm by 100 mm, step size of 2 mm (giving a total of2500 measurements for a single slice along the vertical axis). Therobotic arm performed 10 slices of measurements along the vertical axisthat are centered at the plane where the 2D shape was generated.

FIG. 13 and FIG. 15 show the point cloud distribution of focal pointsfor the straight line and circle shapes, respectively. FIG. 14 and FIG.16 show the spatial distribution of the acoustic radiation force for thetwo 2D shapes (straight line and circle respectively) that are generatedat 12 cm elevation from the ultrasound array surface. The amplitude isnormalized with a maximum force of 2.9 mN. FIG. 14 shows a clearstraight line centered along the y-axis and spread for 100 mm. As shownin FIG. 16, a circle is neatly generated and its diameter is about 70mm. Note that side lobes around the main lobe are generated, in bothcases, that are nearly symmetrical.

3D Tactile Stimulation

For the 3D tactile stimulation, a display of the upper half of a spherewhose center is located at 14 cm elevation, with a radius of 50 mm wasconsidered. The scan volume was adjusted with the followingconfiguration: 100 mm by 100 mm, step size of 2 mm (giving a total of2500 measurements for a single slice making a horizontal plane). Therobotic arm performed 20 slices of measurements along the vertical axisthat started from height 14 cm up to 24 cm, with a step size of 5 mmbetween slices (a total of 20 slices).

FIG. 17 shows the point cloud for the hemisphere object. FIG. 18 showsthe spatial distribution of the acoustic radiation force for thehemisphere object that is centered at 14 cm elevation from theultrasound array surface. The plot in FIG. 18 is based on a thresholdvalue of 2 mN (only forces above 2 mN are displayed to show theconcentration of forces to form the 3D shape).

Simulation

A simulation experiment was conducted in order to confirm whether thegenerated acoustic radiation forces are sufficient for human perceptionof tactile sensation. For the simulation, the pressure fielddistribution and force calculations were computed using a fastobject-oriented C++ ultrasound simulator library called FOCUS that workswith MATLAB, which is executed by a processor or processing circuit. Inother embodiments, a different program may be used including code fromany suitable computer-programming language such as, but not limited to,C, C++, C#, Java, JavaScript, Perl, Python and Visual Basic.

The 3D object used in this simulation was the lower half of a spherelocated centered 100 mm above the transducer array with a radius of 50mm. A graphical representation of the hemisphere is shown in FIG. 19. Inone embodiment, the simulation may include 256 (16×16) transducers, butin other embodiments, the simulation may include another number oftransducers in other configurations or arrangements. In the simulation,the transducer housing was square shaped with a side length of d=10 mm.The resonant frequency was 40 kHz, while the directivity is 80 deg. M×N(=16×16) transducers are used and arranged into a 180×180 mm² rectangle.The following parameters were used to run the simulations describedbelow:

Frequency=40 kHz

β=1.15×10-1 Np/m

C=340 m/s

P0=8.59×10-3 Pa

Voltage=10 Vrms

K=2π/λ=730.603 rd/m

W=2πf

α=2

Medium: an air interface (lossless)

One of ordinary skill in the art will appreciate that a lossless mediumis the air interface. In one embodiment, an ultrasonic wave travels at aspeed of 340 m/s. If the medium were not lossless (i.e., if the mediumchanged), the speed of the ultrasonic wave would change accordingly.

The frequency (e.g., 40 kHz) is the resonance frequency for the selectedultrasonic transducer. The frequency is a constant value based on thehardware transducer in use. Beta is the attenuation coefficient of anultrasonic wave in air interface. Beta is also a constant. The speed ofthe ultrasonic wave (e.g., C=340 m/s) in the air interface. The speed ofthe ultrasonic wave is constant. P0 is the power density of theultrasonic signal at the transducers array surface. P0 is calculated forz=0. Voltage (e.g., 10 Vrms), is the voltage applied to power theultrasonic transducers. Typically, a recommended voltage is provided inthe ultrasonic transducer's datasheet. K is the wavenumber of theultrasonic wave. W is the angular frequency. α is a constant rangingfrom 1 to 2 depending on an amplitude reflection coefficient at anobject surface. If the object surface perfectly reflects the incidentultrasound, α=2, while if it absorbs the entire incident ultrasound,α=1. Assuming that the human hand perfectly reflects the ultrasonicwave, α=2 was selected for the experimental study.

Another configuration parameter that can be controlled through thesimulation is the resolution of tactile 3D object. FIGS. 20A, 20B, and20C illustrate examples of low resolution (e.g., 10 mm) (see FIG. 20A),intermediate resolution (e.g., 5 mm) (see FIG. 20B), and high resolution(e.g., 2 mm) (see FIG. 20C) displays. Note that higher resolutiondisplay takes more time to display the entire 3D object, and thus,require faster hardware to implement. However a low resolution displaymight degrade the quality of user perception of 3D tactile sensation.Usability testing will be conducted in the future work to find theoptimal tradeoff

In the simulation, the following simulation loop was utilized:

% M, N dimensions of the array % w and h are width and height of array %s_x and s_y are x and y spacing array =create_rect_array(M,N,w,h,s_x,s_y) % Loop update rate 1 kHz Loop  %setupcoordinate grid  coord_grid = set_coordinate_grid(delta,  xmin, xmax,ymin, ymax, zmin, zmax);  % Calculate the next point coordinates  point= getNextPoint (3D object)  % focus the array to point  array =get_focus_phase (array, point,)  % calculate pressure field  p =pressure(array, coord_grid, freq,) Repeat Loop

Initially, the geometry of the transducer array was setup with thefollowing dimensions for each transducer (width=1 mm, height=5 mm,edge-to-edge spacing—s_x and s_y=0.5 mm). These values were selected tomaximize the density of ultrasound transducers in the array. Othervalues may be used (other researchers have tried random distribution ofthe transducers spacing. See Gavrilov, L. R. “The Possibility ofGenerating Focal Regions of Complex Configurations in Application to theProblems of Stimulation of Human Receptor Structures by FocusedUltrasound,” Acoustical Physics, Vol. 54, No. 2, pp. 269-278 (2008).

Next a loop that runs at 1 kHz rate was executed. The loop starts bysetting up the coordinate grid to cover the full width of the transducerarray in the x direction and to measure the pressure field in the zdirection. Then the coordinates of the next point on the 3D model areretried and used to generate a focal point at the desired point and holdon for specific time. FIG. 21 shows a simulation that demonstrates theability of ultrasonic transducers to stimulate tactile feeling at adesired point in 3D space.

Table 1 shows simulation results for a combination of 16 differentconfigurations with four resolution levels of display and four layoutconfigurations.

TABLE 1 20% Resolution 10% Resolution 5% Resolution 2% ResolutionUltrasound (2 cm) (1 cm) (0.5 cm) (0.2 cm) Array Avg Std Dev. Avg StdDev Avg Std Dev Avg Std Dev Configuration (mN) (mN) (mN) (mN) (mN) (mN)(mN) (mN) 128X1 3.30 0.38 3.17 0.41 3.14 0.46 3.04 0.54 64X2 3.90 0.963.85 0.94 3.78 0.83 3.70 0.76 32X4 4.33 0.67 4.24 0.62 4.07 0.53 3.830.38 16X8 4.33 0.97 3.96 0.88 3.89 0.66 3.62 0.34

The forces were calculated by utilizing Equation (8) with a cubic unitof 1 mm³. The results of the simulation showed:

-   -   1. In all the considered configurations, the average force        ranged from 4 mN to around 3.5 mN (while standard deviation        ranged from 0.75 mN to 0.5 mN). According to several haptic        studies, this force is comfortably capable of stimulating        tactile sensations on the human skin.    -   2. The higher the display resolution, the smaller the average        forces were. However, the standard deviation of forces mostly        decreased with an increase in the display resolution. This        implies that even though higher variations of displayed forces        were experienced, the average forces decreased. This means that        the average forces have decreased as the resolution of the        displayed object increases. The standard deviation for the        display forces has also increased. Force variations may impact        the quality of a user's experience. Usability testing will be        conducted to investigate and document the quality of experience        associated with the embodiments of the present application. A        well-documented property (called Just-Noticeable-Difference JND)        is known in the literature for tactile stimulation. The JND may        vary based on the tactile stimulation technology used to        generate the forces, as well as the part of the body where the        stimulation is applied (i.e., parts of human body are more        sensitive for tactile sensation than other parts of the human        body).    -   3. The average force increased from a one-dimensional        configuration (1×128) to a two-dimensional configuration (8×16),        with the same total number of ultrasonic transducers. This        implies that using a two dimensional array of ultrasonic        transducers increased the ultrasound forces. However, higher        discrepancies in the magnitude of generated forces were        experienced.

The simulation results showed that the proposed system is capable ofgenerating forces that are within comfortable margins of forces that areperceived by the human skin. Furthermore, the three dimensional tactilefeedback system is capable of generating a sequence of tactile points(that altogether construct a hemisphere in the simulation describedabove) at a 1 kHz update rate.

Comparison with Simulation Results

A similar simulation analysis was conducted to compare the experimentalresults for various tactile objects with the simulation results. Thefollowing parameters were used to run this simulation: frequency=40 kHz,(3=1.15×10-1 Np/m, C=340 m/s, P₀=8.59×10⁻³ Pa, voltage=±12 Volts,K=2π/λ=730.603 rd/m, W=2πf, α=2, and the simulated medium was the airinterface (lossless).

Table 2 summarizes the comparison between experimental and simulationresults for four different shapes: a single point, a straight line, acircle and a hemisphere.

TABLE 2 Simulation Experimental Standard Standard Shapes Average (mN)Deviation Average (mN) Deviation Single Point 2.84 0.11 2.77 0.12Straight line 2.71 0.18 2.68 0.2 Circle 2.69 0.18 2.65 0.19 Hemisphere2.61 0.22 2.66 0.24

For a single point of tactile stimulation, the average tactile forcegenerated is the highest compared to the other objects whereas thestandard deviation is the least. This is explained by the fact thatproducing a single point of tactile stimulation involved minimuminterferences (especially those due to switching from one focal point toanother). It was observed that 3D objects have the least average tactilestimulation and highest standard deviation since interferences arehigher in this case. In all cases, it is clear that all forces(simulation or experimental) are way above what the human skin canperceive and thus the Haptogram system can be used as a haptic displayinterface.

Discussion

The experimental analysis has clearly demonstrated the ability ofHaptogram system to generate perceivable tactile objects (single point,2D shapes and 3D shapes). One tile of the Haptogram display was able togenerate an average force of 2.65 mN (standard deviation of 0.2 mN) forthe 200 focal points that make up the 3D shape, over an elevation rangeof 8 cm to 18 cm.

The Haptogram system has several advantages as a tactile display. Firstof all, it does not provide an undesirable tactile feeling originatingfrom the contact between the device and the skin because the device iscontactless. Second, the temporal bandwidth is broad enough to produce3D tactile feelings by generating a series of tactile focal points andswitching between them at a high speed. The current implementationbreaks down any 3D object into a maximum of 200 focal points.

An audible sound is heard when the Haptogram system is turned on. Theaudible sound may be loud such that it can be heard within severalmeters from the display and it can be annoying for some users.Headphones for ear protection can be used to prevent the user fromhearing the audible sound. There are two sources of the audible sound.One is the envelope of the ultrasound. If 500-Hz modulation is used, the500 Hz audible sound is produced due to the nonlinearity of air, whichis a phenomenon utilized in a directive loudspeaker. The other causewould be the discontinuity of the phases of the driving signals when theposition of the focal point is changed. The inventors believe that theformer source of the audible sound is the dominant one.

There might have been few sources of errors while taking measurementsvia the robotic arm. First of all, the surface of the ultrasound arrayand the scanning surface of the robotic arm may not be perfectlyparallel. This implies that there might be some errors measuring forcesat a particular horizontal slices due to a skewed measurements.Secondly, the sound probe attached to the robotic arm is an ultrasoundtransducer that is not perfectly sensitive to the measurement of forces.Another source of error may have occurred due to additional reflectionnoises as the robotic arm came closer to the surface of thetwo-dimensional array of ultrasound transducers.

Usability Study

A usability study was conducted to investigate how well users canperceive animated 2D shapes displayed by the Haptogram system. Although3D shapes can be presented, the usability study was limited to 2Dobjects. As seen in FIG. 26, four 2D shapes were considered: circle,triangle, line and a plus sign (also referred to as a cross). The shapeswere randomly selected and displayed in a square of 10 cm by 10 cmworkspace with a precision of 1 mm. Users were asked to feel the tactilestimulation through his or her palm. The users were also given noisecancellation headphones to eliminate acoustic noise generated by thesystem or any other auditory source that may distract the user fromrecognizing the tactile shape.

The following configurations were used for tactile stimulation: therendering frequency was set to 10 Hz, while the number of focal pointswas set to 200. The shapes stimuli were displayed in a random order toreduce the bias and ordering effects as much as possible. For eachtrial, the recognition rate and the recognition time were measured. Therecognition rate was defined as the ratio of accurately identifying adisplayed shape over the total number of trials. The recognition timewas defined as the average time it took the user to recognize aparticular shape correctly. The experiment was divided into two blocksof 12 trials for each block, giving participants time to rest from atrial to the next.

A one-tile Haptogram prototype (10×10 transducers) was used in thisexperiment. The experiment software was running on a desktop PC equippedwith an Dual Six CoreXEON E5-2630, 2.3G Hz processor and 32 GB RAM. Ahand-resting stand was designed specifically for this experiment. Activenoise-cancelling headphones were used to cancel out the auditory noise.

Fifteen (15) adult subjects were participated in the experiment; 8 male,7 female, average age of 26 years (standard deviation was 6.60 years).By self-reporting, none of the subjects had any deficiency in theirability to touch. Users were allowed a training session for as much asthey desire to familiarize themselves with the system and the stimulibefore completing the experiment. In total, we collected 360 trials (15participants×2 sessions×12 trials per session).

The experimental task required users to recognize one of the four 2Dshapes by holding their hands on top of the Haptogram display. Theelevation of tactile stimulation was fixed since the user was asked torest his/her palm on a stand tuned at 13 cm elevation (the elevation atwhich the shapes are generated). No audiovisual cues about the displayed2D shapes were given to the participants. Each trial started when theparticipant hit the return button of the keyboard in front of them.Participants were instructed to feel tactile shapes on his/her palm andrespond as fast and accurate as possible. The participant responded bykeying a corresponding number (1 for circle, 2 for plus sign, 3 forline, and 4 for triangle). Each shape was displayed continuously for amaximum of 30 seconds. If the participant failed to give a responsewithin the 30 seconds, then the trial was cancelled. As soon as theparticipant gave a response, the response time was recorded along withwhether the selection was correct or not. Next, the participant wasgiven some time to rest before proceeding to the next trial.

The overall recognition rate for all subjects was well above the chancelevel (average of 59.44%, standard deviation of 12.75%). The overallrecognition time had an average of 13.87 seconds and a standarddeviation of 3.92 seconds. Looking at the recognition rate for eachshape, the plus sign shape was the easiest to recognize, while the lineshape was the most difficult to recognize. This could be because theline shape is easier to confuse with the circle and/or triangle shapes.The average recognition rate for all the shapes, along with the standarddeviations, are shown in FIG. 27. It was also observed that oneparticipant never got the triangle shape.

As for the recognition time, results show that the plus sign shape hadthe largest recognition time compared to other shapes, while the lineshape had the lowest recognition time (FIG. 28). However, thedifferences are not that significant to derive definite conclusionsabout the recognition speed for each shape; more or less all shapes havesimilar recognition times.

In an effort to recognize which shapes were the most confusing, the datawas analyzed to find errors across combinations of shapes. Table 3 showsthe results (e.g., 14.45% is the percentage of confusing the triangleshape as a circle, and so on).

TABLE 3 Confusion Matrix Across the Four Shapes Circle Triangle LinePlus sign Circle 62.23% 6.67% 26.67% 4.45% Triangle 14.45% 65.56% 12.23%7.78% Line 51.12% 2.23% 44.45% 2.23% Plus sign 6.67% 10.00% 7.78% 75.56%It seems that the line shape was mostly confused with circle (51.12% ofthe times line was confused and recognized as circle). The circle wasalso highly confused with the line (26.67%). The most clearly perceivedshape was the plus sign; it had around 75% recognition rate, and wasconfused the least among the four shapes.

Even though it would be hard to derive statistically valid conclusions,a considerable improvement in performance from session to session wasobserved. In particular, there was a 14% improvement in the recognitionrate and a 12% decrease in the recognition time. This clearly shows thatusers have quickly learned how to use the device and effectivelyperceive 2D shapes.

The embodiments described above are directed to a Haptogram systemcapable of displaying 3D tactile objects. The Haptogram system enablesusers to feel virtual objects without physical touches on any displaydevices. The Haptogram system may include one tile having atwo-dimensional array of ultrasound transducers (e.g., 10 by 10transducers). The ability of the Haptogram system to display varioustypes of objects (single point, 2D shapes and 3D shapes) was validatedin the experiments described above. As a result, it was confirmedthat 1) humans can feel a localized focal point clearly (average forcegenerated is 2.9 mN which is well above what human skin can perceive)and 2) the Haptogram system is highly precise in displaying tactileobjects. The measured force along all displayed objects (single point,2D objects and 3D objects) had an average of 2.65 mN and a standarddeviation of 0.2 mN. In other words, the results showed that alldisplayed tactile objects are perceivable by the human skin (an averageof 2.65 mN, standard deviation of 0.2 mN, for the 200 focal points thatmake up the 3D shape, over an elevation range of 8 cm to 18 cm).

In one application, the three dimensional tactile feedback system can beintegrated, for example, with a graphic hologram to build a completemultimodal immersive experience (auditory, visual, and tactilefeedback). See FIG. 22. In other words, the Haptogram system may be animmersive multimodal system where subjects can touch graphics holograms.The Haptogram display may be synchronized with a hologram display,and/or a hand motion tracking device such as Leap Motion™ or MicrosoftKinect™. One of ordinary skill in the art would appreciate otherapplications of the three dimensional tactile feedback system describedabove.

The foregoing description of embodiments of the invention has beenpresented for purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed, and modifications and variations are possible in light of theabove teachings or may be acquired from practice of the invention. Theembodiments were chosen and described in order to explain the principalsof the invention and its practical application to enable one skilled inthe art to utilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplated. Othersubstitutions, modifications, changes and omissions may be made in thedisclosure's operating conditions and arrangement of the embodimentswithout departing from the scope of the present invention.

As utilized herein, the terms “approximately,” “about,” “substantially”,and similar terms are intended to have a broad meaning in harmony withthe common and accepted usage by those of ordinary skill in the art towhich the subject matter of this disclosure pertains. It should beunderstood by those of skill in the art who review this disclosure thatthese terms are intended to allow a description of certain featuresdescribed and claimed without restricting the scope of these features tothe precise numerical ranges provided. Accordingly, these terms shouldbe interpreted as indicating that insubstantial or inconsequentialmodifications or alterations of the subject matter described and claimedare considered to be within the scope of the invention as recited in theappended claims.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for thesake of clarity.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software embodied on a tangible medium, firmware, or hardware,including the structures disclosed in this specification and theirstructural equivalents, or in combinations of one or more of them.Embodiments of the subject matter described in this specification can beimplemented as one or more computer programs, i.e., one or more modulesof computer program instructions, encoded on one or more computerstorage medium for execution by, or to control the operation of, dataprocessing apparatus. Alternatively or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially-generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate components or media(e.g., multiple CDs, disks, or other storage devices). Accordingly, thecomputer storage medium may be tangible and non-transitory.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus or processingcircuit on data stored on one or more computer-readable storage devicesor received from other sources.

The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors or processing circuitsexecuting one or more computer programs to perform actions by operatingon input data and generating output. The processes and logic flows canalso be performed by, and apparatus can also be implemented as, specialpurpose logic circuitry, e.g., an FPGA or an ASIC.

Processors or processing circuits suitable for the execution of acomputer program include, by way of example, both general and specialpurpose microprocessors, and any one or more processors of any kind ofdigital computer. Generally, a processor will receive instructions anddata from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for performing actionsin accordance with instructions and one or more memory devices forstoring instructions and data. Generally, a computer will also include,or be operatively coupled to receive data from or transfer data to, orboth, one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device (e.g., a universalserial bus (USB) flash drive), to name just a few. Devices suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display), OLED (organic light emitting diode), TFT (thin-filmtransistor), plasma, other flexible configuration, or any other monitorfor displaying information to the user and a keyboard, a pointingdevice, e.g., a mouse trackball, etc., or a touch screen, touch pad,etc., by which the user can provide input to the computer. Other kindsof devices can be used to provide for interaction with a user as well;for example, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including acoustic,speech, or tactile input. In addition, a computer can interact with auser by sending documents to and receiving documents from a device thatis used by the user; for example, by sending web pages to a web browseron a user's client device in response to requests received from the webbrowser.

What is claimed is:
 1. A three dimensional tactile feedback devicecomprising: an array of ultrasonic transducers; a control deviceconfigured to control the array of ultrasonic transducers; wherein thearray of ultrasonic transducers are configured to project discretepoints of tactile feedback in three dimensional space; wherein thetactile feedback device is configured to continuously switch thediscrete points of tactile feedback at a frequency at which a human iscapable of perceiving tactile stimulation in order to produce tactilestimulation of a three dimensional object in free space.
 2. The threedimensional tactile feedback device of claim 1, wherein the array ofultrasonic transducers comprises a plurality of tile units, each tileunit having a two-dimensional array of ultrasonic transducers.
 3. Thethree dimensional tactile feedback device of claim 2, wherein each tilecomprises a ten by ten two-dimensional array of ultrasonic transducers.4. The three dimensional tactile feedback device of claim 1, furthercomprising a point cloud representation component including a pluralityof 3D point cloud representations, each 3D point cloud representationincluding a plurality of discrete points that correspond to a singlepoint, a two dimensional object or a three dimensional object, whereinthe control device is configured to input a 3D point cloudrepresentation corresponding to a desired single point, two dimensionalobject or three dimensional object, and project the discrete points ofthe 3D point cloud representation into three dimensional space toproduce tactile stimulation of the desired single point, two dimensionalobject or three dimensional object in free space.
 5. The threedimensional tactile feedback device of claim 1, further comprising agraphical user interface.
 6. The three dimensional tactile feedbackdevice of claim 1, further comprising an amplifier circuit configured toreceive and amplify synchronized pulse signals generated by the controldevice to stimulate the array of ultrasonic transducers.
 7. The threedimensional tactile feedback device of claim 1, wherein the controldevice comprises: a primary memory configured to store informationrelated to generating each focal point; and a secondary memoryconfigured to store a sequence of index numbers, wherein each indexnumber corresponds to the information stored in the primary memory thatis related to a particular focal point.
 8. The three dimensional tactilefeedback device of claim 7, further comprising a focal point calculationcomponent configured to calculate: 1) a distance between each focalpoint and each transducer in the array of ultrasonic transducers, and 2)a stimulation timing of each transducer, wherein the calculateddistances and stimulation timings are stored in the primary memory asthe information related to generating each focal point.
 9. The threedimensional tactile feedback device of claim 7, wherein the informationstored by the primary memory includes at least one of a timing queue foreach of the ultrasonic transducers, a cycle length, a pulse length, anda pre-scalar clock divider value.
 10. The three dimensional tactilefeedback device of claim 7, wherein the control device further comprisesa memory controller, a cycle controller, and a pulse generator, thememory controller configured to extract the information stored in theprimary memory that is related to the particular focal point and providethe information to the cycle controller and the pulse generator.
 11. Thethree dimensional tactile feedback device of claim 10, wherein the cyclecontroller is configured to determine a width of a period of a pulseused to stimulate the array of ultrasonic transducers to projectdiscrete points of tactile feedback in three dimensional space.
 12. Thethree dimensional tactile feedback device of claim 10, wherein the pulsegenerator is configured to generate pulses for each ultrasonictransducer at an appropriate time in order to project the discretepoints of tactile feedback related to the particular focal point inthree dimensional space.
 13. A method of producing a tactile stimulationof a three dimensional object comprising: providing a three dimensionaltactile feedback device comprising: a two-dimensional array ofultrasonic transducers; and a control device configured to control thearray of ultrasonic transducers; projecting discrete points of tactilefeedback in three dimensional space with the array of ultrasonictransducers; and continuously switching the discrete points of tactilefeedback at a frequency at which a human is capable of perceivingtactile stimulation in order to produce tactile stimulation of a threedimensional object in free space.
 14. The method of claim 13, whereinprojecting discrete points of tactile feedback in three dimensionalspace comprises importing information related to a particular focalpoint from a primary memory of the control device.
 15. The method ofclaim 14, further comprising calculating 1) a distance between eachfocal point and each transducer in the array of ultrasonic transducers,and 2) a stimulation timing of each transducer, wherein the calculateddistances and stimulation timings are stored as the information relatedto a particular focal point in the primary memory.
 16. The method ofclaim 14, wherein the information comprises at least one of a timingqueue for each of the ultrasonic transducers, a cycle length, a pulselength, and a pre-scalar clock divider value.
 17. The method of claim14, wherein projecting discrete points of tactile feedback in threedimensional space further comprises extracting the information stored inthe primary memory related to the particular focal point and providingthe information to the control device.
 18. The method of claim 17,wherein projecting discrete points of tactile feedback in threedimensional space further comprises determining a width of a period of apulse used to stimulate the array of ultrasonic transducers to projectdiscrete points of tactile feedback in three dimensional space.
 19. Themethod of claim 18, wherein projecting discrete points of tactilefeedback in three dimensional space further comprises to generatingpulses for each ultrasonic transducer at an appropriate time in order toproject the discrete points of tactile feedback related to theparticular focal point in three dimensional space.
 20. The method ofclaim 19, wherein projecting discrete points of tactile feedback inthree dimensional space further comprises amplifying synchronized pulsesignals generated by the control device to stimulate the array ofultrasonic transducers.
 21. A computer-implemented machine for threedimensional tactile feedback, comprising: a processor; a threedimensional tactile feedback device comprising: a two-dimensional arrayof ultrasonic transducers; a control device configured to control theultrasonic transducers; a tangible computer-readable medium operativelyconnected to the processor and including computer code configured to:project discrete points of tactile feedback in three dimensional spacewith the ultrasonic transducers; and continuously switch the discretepoints of tactile feedback at a frequency at which a human is capable ofperceiving tactile stimulation in order to produce tactile stimulationof a three dimensional object in free space.